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DESCRIPTION 



Image Pick-up Apparatus and Image Pick-up Method 
Technical Field 

The present invention relates to an image pick-up apparatus and an 
image pick-up method which are adapted for picking up image of object, and 
more particularly to an image pick-up apparatus and an image pick-up method 
which are adapted for performing image picknip operation in a manner 
suitable with respect to image pick-up sensitivity. 

Further, this Application claims priority of Japanese Patent 
Application No. 2002-375423, filed on December 25, 2002, the entirety of 
which is incorporated by reference herein. 

Background Art 

In recent years, image pick-up apparatuses (digital cameras, and/or 
color scanners, etc.) and image processing softwares which are oriented to 
consumer have been popularized. Users who themselves edit, by image 
processing software, image generated by picking up an image of object by the 
image pick-up apparatus are being increased. Moreover, request with respect 
to picture quality of image which has been picked up by the image pick-up 
apparatus has become strong. Ratio of users who mention that picture 
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quality is high as main condition in purchasing the image pick-up apparatus is 
being increased. Here, general configuration of the image pick-up apparatus 
will be described below. 

In the image pick^ip apparatus, there is used, e.g., color filter 1 for 
three primary colors of RGB as shown in FIG. 1. In this example, color filter 
1 is constituted by the so-called Bayer arrangement with four filters in total of 
two G filters serving to allow only light of green (G) to be transmitted 
therethrough, one R filter serving to allow only light of red (R) to be 
transmitted therethrough and one B filter serving to allow only light of blue 
(B) to be transmitted therethrough being as minimum unit as indicated by 
single dotted lines of FIG. 1 . 

FIG. 2 is a block diagram showing a configuration example of a signal 
processing unit 1 1 for implementing various processing to RGB signals which 
have been acquired by CCDs (Charge Coupled Devices) having the RGB 
color filter 1. 

An offset correction processing unit 21 serves to remove offset 
component included in an image signal which has been delivered from a front 
end 13 for implementing a predetermined processing to signals which have 
been acquired by the CCD image pick-iip devices to output an image signal 
thus obtained to a white balance correction processing unit 22. The white 
balance correction processing unit 22 serves to correct balance of respective 
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colors on the basis of color temperature of the image signal which has been 
delivered from the offset correction processing unit 21 and differences of 
sensitivities of respective filters of the color filter 1. The color signal 
acquired after undergone correction by the white balance correction 
processing unit 22 is outputted to a gamma (Y)-correction processing unit 23. 
The gamma-correction processing unit 23 performs gamma-correction with 
respect to a signal which has been delivered from the white balance correction 
processing unit 22 to output a signal thus acquired to a vertical direction 
simultaneous-izing (synchronizing) processing unit 24. At the vertical 
direction simultaneous-izing processing unit 24, delay elements are provided. 
Shifts of times in vertical direction of signal which have been delivered from 
the gamma-correction processing unit 23 are simultaneous-ized 
(synchronized). 

A RGB signal generation processing unit 25 performs interpolation 
processing for interpolating color signal delivered from the vertical direction 
simultaneous-izing processing unit 24 into phase of the same space, noise 
rejection (removal) processing for rejecting or removing noise component of 
signal, filtering processing for limiting signal band, and high frequency band 
correction processing for correcting high frequency band component of the 
signal band, etc. to output RGB signals thus obtained to a luminance signal 
generation processing unit 26, and. a color difference signal generation 
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processing unit 27. 

The luminance signal generation processing unit 26 serves to 
synthesize, at a predetermined synthesis ratio, RGB signals which have been 
delivered from the RGB signal generation processing unit 25 to generate a 
luminance signal (Y). The color difference signal generation processing unit 
27 serves to similarly synthesize, at a predetermined synthesis ratio, RGB 
signals which have been delivered from the RGB signal generation processing 
unit 25 to generate color difference signals (Cb, Cr). The luminance signal 
(Y) generated by the luminance signal generation processing unit 26 and the 
color difference signals (Cb, Cr) generated by the color difference signal 
generation processing unit 27 are outputted to a monitor provided at the 
outside of the signal processing unit 11, for example. 

In a manner as stated above, it is generally performed to implement 
gamma^rocessing to an original signal and to conduct thereafter image 
processing (linear matrix processing) by linear transform processing. 

In the image pick-up apparatus as described above, there are instances 
where since when image of object is picked up to generate image, way of 
seeing varies depending upon . visual environment at the time of observation, 
reproduction into desired color may not be performed. This is the 
phenomenon taking place in the case where color rendering characteristics of 
light source at the time of image pick-up operation (hereinafter referred to as 



image pick-up light source) and light source at the time of observation 
(hereinafter referred to an observation light source) are greatly different from 
each other. In view of the above, there is proposed, in the Japanese Patent 
Application Laid Open No. 2002-142231 publication, etc., a method of 
satisfactorily performing color reproduction even in the case where image is 
reproduced by the observation light source having color rendering 
characteristic different from that of the image pick-^p light source. 
Moreover, characteristics of spectral sensitivities are shown in FIGS. 3 and 4. 
Curve LI of FIG. 3 indicates spectral sensitivity of R, curve L2 indicates 
spectral sensitivity of G, and curve L3 indicates spectral sensitivity of B. 
Further, curve LU of FIG. 4 indicates spectral sensitivity of R, curve L12 
indicates spectral sensitivity of G, and curve LI 3 indicates spectral sensitivity 
of B. 

On the other hand, in the image pick-up apparatus as described above, 
addition of the linear matrix processing is performed, and addition of color 
filter of the image pick-up device is further performed, thereby making it 
possible to dramatically improve color reproduction characteristic 
(reproducibility). In this instance, when coefficients of a linear matrix used 
is determined so that color difference is simply minimized, there are instances 
where noise may be increased as compared to the prior art. 
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Disclosure of the Invention 

An object of the present invention is to provide an image pick-up 
apparatus and an image pick-up method which can perform linear matrix 
processing using coefficients in which color reproduction characteristic and 
noise reduction characteristic are taken into consideration in dependency upon 
image pick environment and/or image pick-up condition, etc. 

The image pick-up apparatus according to the present invention is 
directed to an image pick-up apparatus including an image pick-up unit 
comprised of color filters having different spectral characteristics and serving 
to pick up image of object, which comprises: adjustment means for adjusting 
color reproduction value and noise value representing noise feeling; matrix 
coefficient determination means for determining matrix coefficients on the 
basis of adjustment of the adjustment means; and matrix transform processing 
means for performing matrix transform processing with respect to an image 
which has been picked up at the image pick-Hip device unit on the basis of the 
matrix coefficients. 

In addition, the image pick-^p method according to the present 
invention is directed to an image pick-up method of picking up an image of 
object by an image pick-up apparatus including an image pick-up unit 
comprised of color filters having different spectral characteristics and serving 
to pick up image of object, which includes: a first step of adjusting color 
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reproduction value and noise value representing noise feeling; a second step 
of determining matrix coefficients on the basis of adjustment of the first step; 
and a third step of performing matrix transform processing with respect to an 
image which has been picked up at the image pick-up unit on the basis of the 
matrix coefficients. 

Still further objects of the present invention and practical merits 
obtained by the present invention will become more apparent from the 
description of the embodiments which will be given below. 

Brief Description of the Drawings 

FIG. 1 is a view showing an example of conventional RGB color 

filter. 

FIG. 2 is a block diagram showing a configuration example of signal 
processing unit provided in a conventional image pick-up apparatus. 

FIG. 3 is a view showing an example of spectral sensitivity 
characteristic. 

FIG. 4 is a view showing another example of spectral sensitivity 
characteristic. 

FIG. 5 is a block diagram showing a configuration example of an 
image pick-up apparatus to which the present invention is applied. 

FIG. 6 is a view showing an example of color filter for four colors 
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provided in the image pick-Hip apparatus to which the present invention is 
applied. 

FIG. 7 is a view showing an example of spectral luminous efficacy 

curve. 

FIG. 8 is a view showing feature of evaluation coefficients. 

FIG. 9 is a block diagram showing a configuration example of camera 
system LSI that the image pick-iip apparatus to which the present invention is 
applied has. 

FIG. 10 is a block diagram showing a configuration example of signal 
processing unit of FIG. 9. 

FIG. 11 is a flowchart for explaining preparation processing of image 
processing unit. 

FIG. 12 is a flowchart for explaining the detail of processing for 
determining color filter for four colors of step SI of FIG. 11, 

FIG. 13 is a view showing an example of virtual curve. 

FIGS. 14(A) to 14(C) are views showing examples of UMG value 
every filter. 

FIG. 15 is a view showing an example of spectral sensitivity 
characteristic of color filter for four colors. 

FIG. 16 is a flowchart for explaining the detail of linear matrix 
determination processing of step S2 of FIG. 11. 



FIG. 17 is a view showing an example of evaluation result of color 
difference. 

FIG. 18 is a view showing chromaticity of a predetermined object by 
color filter for four colors. 

FIG. 19 is a view showing another example of color filter for four 
colors provided in the image pick-^p apparatus to which the present invention 
is applied. 

FIG. 20 is a flowchart showing that linear matrix coefficients M are 
adaptively determined, 

FIG. 21 is a view showing the state of change of noise reduction 
characteristic index when color reproduction characteristic index is changed. 

FIG. 22 is a view showing an example of spectral sensitivity 
characteristic of color filter for four colors. 

FIG. 23 is a view showing histogram of image. 

Best Mode for Carrying Out the Invention 

FIG. 5 is a block diagram showing a configuration example of an 
image picknip apparatus to which the present invention is applied. 

In the image pick-up apparatus shown in FIG. 5, a color filter for 
discriminating between four kinds of colors (rays of light) is provided at the 
front face (the face opposite to lens 42) of an image sensor 45 comprised of 
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CCDs (Charge Coupled Devices), etc. It is to be noted that, in the image 
pick-up apparatus, the color filter provided at the image sensor 45 of FIG. 5 is 
caused to be a filter 61 for four colors shown in FIG. 6. 

The color filter 61 for four colors is constituted with four filters in 
total of R filter for allowing only red light to be transmitted therethrough, B 
filter for allowing only blue light to be transmitted therethrough, Gl filter for 
allowing only green light of the first wavelength band to be transmitted 
therethrough, and G2 filter for allowing green light of the second wavelength 
band, which has high correlation with respect to the Gl filter, to be 
transmitted therethrough being as the minimum unit as indicated by single 
dotted lines of FIG. 6. Moreover, the Gl filter and the G2 filter are arranged 
at positions diagonal to each other within the minimum unit thereof. 

As described in detail later, the number of kinds of colors of image 
acquired by the image sensor 45 is caused to be four, and color information to 
be acquired is increased, thereby making it possible to more precisely 
represent color as compared to the case where only three kinds of colors 
(RGB) are acquired. Further, it becomes possible to improve the fact that 
color seen as different color by the eye is reproduced into different color, and 
color seen as the same color is reproduced into the same color 
("discriminating characteristic of color"). 

It is to noted that the eye of the human being is sensitive to luminance 
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as understood from spectral luminous efficacy curve shown in FIG. 7. 
Accordingly, in the color filter 61 for four colors shown in FIG. 6, there is 
supplemented color filter of G2 having spectral sensitivity characteristic close 
to spectral luminous efficacy curve in which more precise luminance 
information is acquired so that gradation of luminance can be raised, and 
image close to way of seeing of the eye can be reprroduced (there is 
supplemented green G2 filter newly determined with respect to filters of R, 
Gl, B corresponding to R, G, B of FIG. 1). 

Moreover, as filter evaluation coefficient used in determining color 
filter 61 for colors, there is used UMG (Unified Measure of Goodness) which 
is coefficient in which, e.g., both "color reproduction charcteristic" and "noise 
reduction characteristic" are taken into consideration. 

In the evaluation using UMG, its evaluation value does not become 
high in the case where filter to be evaluated simply satisfies the router 
condition, but overlap of spectral sensitivity distributions of respective filters 
is also taken into consideration. Accordingly, noise can be more reduced as 
compared to the case of the color filter evaluated by utilizing q-factor, 
|Li-f actor or FOM (Figure of Merit). Namely, by evaluation using UMG, there 
is selected filter in which spectral sensitivity characteristics of respective 
filters overlap with each other to a certain degree, but substantially all 
characteristics do not overlap as in the case of the characteristic of R and the 
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characteristic of G of FIG. 4. For this reason, even in the case where 
respective color signals are amplified for the purpose of separation of colors, 
it is unnecessary to allow amplification factor to be so large. Followed by this, 
it is suppressed that noise component is amplified. 

FIG. 8 is a view showing feature of respective filter evaluation 
coefficients. It is to be noted that FIG. 8 shows the matter relating to the 
number of filters which can be evaluated at a time, whether or not spectral 
reflection factor of object is taken into consideration, and whether or not 
reduction of noise is taken into consideration with respect to respective 
evaluation coefficients. 

As shown in FIG. 8, q-factor indicates that the number of filters which 
can be evaluated at a time is only "one", and spectral reflection factor of 
object and reduction of noise are not taken into consideration. Moreover, 
|i-factor indicates that while plural filters can be evaluated at a time, spectral 
reflection factor of object and reduction of noise are not taken into 
consideration. Further, FOM indicates that while plural filters can be 
evaluated at a time and spectral reflection factor of object is taken into 
consideration, reduction of noise is not taken into consideration. 

On the contrary, UMG used in determining color filter 61 for four 
colors indicates that plural filters can be evaluated at a time, and spectral 
reflection factor of object is taken into consideration and reduction of noise is 
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also taken into consideration. 

It is to be noted that the detail of q-factor is disclosed in 'H.EJ. 
Neugebauer "Quality Factor for Filters Whose Spectral Transmittances are 
Different from Color Mixture Curves, and Its Application to Color 
Photography" JOURNAL OF THE OPTICAL SOCIETY OF AMERICA, 
VOLUME 46, NUMBER 10', and the detail of |Li-factor is disclosed in 'RL. 
Vora and HJ. Trussell, "Measure of Goodness of a set of color-scanning 
filters", JOURNAL OF THE OPTICAL SOCIETY OF AMERICA, 
VOLUME 10, NUMBER 1\ Moreover, the detail of FOM is disclosed in 'G. 
Sharma and HJ. Trussell, "Figures of Merit for Color Scanners, IEEE 
TRANSACTION ON IMAGE PROCESSING, VOLUME 6', and the detail of 
UMG is disclosed in 'S-Quan, N.Ohta, and N.Katoh, "Optimal Design of 
Camera Spectral Sensitivity Functions Based on Practical Filter Components", 

cic, 20or. 

Returning to the explanation of FIG. 5, the microcomputer 41 controls 
the entire operation in accordance with predetermined control programs. For 
example, the microcomputer 41 performs exposure control by an iris 43, 
opening/closing control of a shutter 44, control of electronic shutter of a TG 
(Timing Generator) 46, gain control at a firont end 47, mode control of a 
camera system LSI (Large Scale Integrated Circuit) 48, and parameter control, 
etc. 
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The iris 43 adjusts passing of light (iris) converged by a lens 42 to 
control light quantity which is taken thereinto by image sensor 45. The shutter 
44 controls passing of light which has been converged by the lens 42 on the 
basis of instruction of the microcomputer 41. 

The image sensor 45 further includes image picknup device comprised 
of CCDs or CMOSs (Complementary Metal Oxide Semiconductors), and 
serves to convert rays of light incident through color filter 61 for four colors 
formed at the firont face of the image pickup device into electric signals to 
output four kinds of color signals (R signal, Gl signal, G2 signal, B signal) to 
the front end 47. At the image sensor 45, color filter 61 for four colors of FIG. 
6 is provided, so that from rays of light incident through the lens 42, 
components of wavelengths of respective bands of R, Gl, G2, B are extracted. 
It is to be noted that the detail thereof will be described later with reference to 
FIG. 15. 

The front end 47 implements correlating double sampling processing 
for removing noise component, gain control processing and digital conversion 
processing, etc. to color signal delivered from the image sensor 45. The image 
data obtained after undergone various processing by the front end 47 is 
outputted to the camera system LSI 48. 

As described in detail later, the camera system LSI 48 performs 
various processing with respect to the image data delivered from the front end 
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47 to generate, e.g., luminance signal and color signal, to output those signals 
to image monitor 50, and to allow the image monitor 50 to display image 
corresponding to the signals. 

An image memory 49 is comprised of, e.g., DRAM (Dynamic 
Random Access Memory), or SDRAM (Synchronous Dynamic Random 
Access Memory), etc., and is suitably utilized when the camera system LSI 48 
performs various processing. An external memory medium 51 comprised of 
semiconductor memory or disc, etc. is detachably constituted with respect to, 
e.g., the image pick-^p apparatus of FIG. 5, so that image data compressed by 
JPEG (Joint Photographic Expert Group) format by the camera system LSI 48 
are stored thereinto. 

The image monitor 50 is comprised of, e.g., LCD (liquid Crystal 
Display), etc., and serves to display picked-up images and/or various menu 
pictures, etc. 

FIG. 9 is a block diagram showing a configuration example of the 
camera system LSI 48 shown in FIG. 5. Respective blocks constituting the 
camera system LSI 48 are controlled by the microcomputer 41 shown in FIG. 
5 through a microcomputer interface (I/F) 73. 

A signal processing unit 71 performs various processing such as 
interpolation processing, filtering processing, matrix operation processing, 
luminance signal generation processing and/or color difference signal 
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generation processing, etc. with respect to four kinds of color information 
delivered from the front end 47 to output, e.g., generated image signal to the 
image monitor 50 through a monitor interface 77. 

An image detecting unit 72 performs detection processing such as 
auto-focus, auto-exposure, and/or auto^hite balance, etc. on the basis of 
output of the front end 47 to output the result thereof to the microcomputer 41 
as occasion demands. 

A memory controller 75 controls transmission/reception of data 
between processing blocks mutually, or transmission/reception of data 
between a predetermined processing block and the image memory 49 to 
output, e.g., image data delivered from the signal processing unit 71 to the 
image memory 49 through the memory interface 74 and to allow the image 
memory 49 to store such image data. 

An image compression/decompression (extraction) unit 76 compresses, 
e.g., image data delivered from the signal processing unit 71 by the JPEG 
format to output data thus obtained to the external memory medium 51 
through the microcomputer interface 73 to allow the external memory medium 
51 to store such data. The image compression/decompression unit 76 further 
decompresses or extracts (expands) the compressed data which has been read 
out from the extemal memory medium 51 to output the data thus obtained to 
the image monitor 50 through the monitor interface 77. 
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FIG. 10 is a block diagram showing the detailed configuration 
example of the signal processing unit 71 shown in FIG. 9. Respective blocks 
constituting the signal processing unit 71 are controlled by the microcomputer 
41 through the microcomputer interface 73. 

An offset correction processing unit 91 removes noise component 
(offset component) included in image signal delivered from the front end 47 to 
output the image signal thus obtained to a white balance correction processing 
unit 92. The white balance correction processing unit 92 serves to correct 
balance of respective colors on the basis of color temperatuere of image signal 
delivered from the offset correction processing unit 91 and differences 
between sensitivities of respective filters of the color filter 61 for four colors. 
A color signal acquired after undergone correction by the white balance 
correction processing unit 92 is outputted to a vertical direction 
simultaneous4zing (synchronizing) processing unit 93. At the vertical 
direction simultaneous-izing processing unit 93, delay elements are provided. 
Thus, shifts of time in vertical direction of signals outputted from the white 
balance correction processing unit 92 (hereinafter referred to as RG1G2B 
signals) are simulataneous-ized or synchronized (corrected). 

A signal generation processing unit 94 performs interpolation 
processing for interpolating color signal of 2 ^ 2 pixels of the minimum unit 
of RG1G2B signals delivered from the. vertical direction simulataneous-izing 
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processing unit 93 into phase of the same space, noise removal processing for 
removing noise component of signal, filtering processing for limiting signal 
band, and high frequency band correction processing for correcting high 
frequency band component of signal band, etc. to output the RG1G2B signals 
thus obtained to a linear matrix processing unit 95. 

The linear matrix processing unit 95 performs operation (computation) 
of the RG1G2B signals by the formula (1) on the basis of predetermined 
linear matrix coefficients (matrix of 3 x 4) to generate RGB signals of three 
colors. 
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An R signal generated by the linear mattix processing unit 95 is 
outputted to a gamma^orrection processing unit 96-1, a G signal is outputted 
to a gamma correction processing unit 96-2, and a B signal is outputted to a 
ganraia-correction processing unit 96-3. 

The gamma-correction processing units 96-1 to 96-3 perform ganmnia 
correction with respect to respective signals of RGB signals which have been 
outputted from the linear matrix processing unit 95 to output RGB signals 
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thus acquired to a luminance (Y) signal generation processing unit 97 and a 
color difference (C) signal generation processing unit 98. 

The luminance signal generation processing unit 97 synthesizes, at a 
predetermined synthesis ratio, RGB signals delivered from the gamma 
correction processing units 96-1 to 96-3 in accordance with, e.g., the formula 
(2) to generate a luminance signal (Y). 

Y = 0.2126R + 0.7152G + 0.0722B (2) 



A color difference signal generation processing unit 98 similarly 
synthesizes, at a predetermined synthesis ratio, RGB signals delivered from 
the gamma correction processing units 96-1 to 96-3 to generate a color 
difference signal (C) to output the color difference signal (C) thus generated to 
a band limit thinning processing unit 99. The band limit thinning processing 
unit 99 generates color difference signals (Cb, Cr) on the basis of the color 
difference signal (C). It is to be noted that in a signal obtained by single chip 2 
X 2 color coding, band of color information generally does not exist to such a 
degree that such band of color information exists in luminance signal. 
Accordingly, the band limit thinning processing unit 99 performs band limit 
processing and thinning processing with respect to color difference signal (C) 
delivered from the color signal generation processing unit 98 to thereby 



20 



reduce color information data and to generate color difference signals (Cb, 
Cr). 

A luminance signal (Y) generated by the luminance signal generation 
processing unit 97, and a color difference signal (C) generated by the color 
difference signal generation processing unit 98, or color difference signals (Cb, 
Cr) generated by the band limit thinning processing unit 99 are outputted to 
the image monitor 50 through, e.g., monitor interface 77 shown in FIG. 9. 

In the image picknip apparatus having the configuration as described 
above, in the case where photographing of image is instructed, the 
microcomputer 41 controls TG46 to take image thereinto by image sensor 45. 
Namely, rays of light of four colors are transmitted by color filter 61 for four 
colors formed at the front face of image pickup devices such as CCD, etc. 
(hereinafter referred to as CCD image pick-up device) constituting the image 
sensor 45. The rays of light thus transmitted are taken in by the CCD image 
pick-up devices. The rays of light which have been taken in by the CCD 
image pick-up device are converted into color signals of four colors. The 
color signals thus obtained are outputted to the front end 47. 

The front end 47 implements correlating double sampling processing 
for removing noise component, gain control processing, and digital convertion 
processing, etc. with respect to the color signal delivered from the image 
sensor 45 to output image data thus obtained to the camera system LSI 48. 
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At the signal processing unit 71 of the camera system LSI 48, offset 
component of color signal is removed by the offset correction processing unit 
91, and balance of respective colors is corrected on the basis of color 
temperature of image signal and differences betv^een sensitivities of 
respective filters of the color filter 61 for four colors by the white balance 
correction processing unit 92. 

Moreover, shifts of time in vertical direction of signals corrected by 
the white balance correction processing unit 92 are simultaneous4zed or 
synchronized (corrected) by the vertical direction simulataneous-izing 
processing unit 93, and interpolation processing for interpolating color signal 
of 2 X 2 pixels of the minimum unit of RG1G2B signals delivered from the 
vertical direction simultaneous^zing processing unit 93 into the phase of the 
same space, noise removal processing for removing noise component of signal, 
filtering processing for limiting signal band, and high frequency band 
correction processing for correcting high frequency component of signal band, 
etc. are performed by the signal generation processing unit 94. 

Further, at the linear matrix processing unit 95, signals generated by 
the signal generation processing unit 94 (RG1G2B signals) are transformed on 
the basis of predetermined linear matrix coefficients M (matrix of 3 ^ 4). 
Thus, RGB signals of three colors are generated. An R signal generated by 
the linear matrix processing unit 95 is outputted to , gamma-correction 
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processing unit 96-1, a G signal is outputted to the gamma-<:oiTection 
processing unit 96-2, and a B signal is outputted to the gamma-correction 
processing unit 96-3. 

Gamma-correction is performed, by the ganmia correction processing 
units 96-1 to 96-3, with respect to respective signals of RGB signals obtained 
by processing of the linear matrix processing unit 95. The RGB signals thus 
acquired are outputted to the luminance signal generation processing unit 97 
and the color difference signal generation unit 98. At the luminance signal 
generation processing unit 97 and the color difference signal generation 
processing unit 98, respective signals of R signal, G signal and B signal which 
are delivered from the ganraia^orrection processing units 96-1 to 96-3 are 
synthesized at a predetermined synthesis ratio. Thus, luminance signal (Y) 
and color difference signal (C) are generated. The luminance signal (Y) 
generated by the luminance signal generation processing unit 97, and the color 
difference signal (C) generated by the color difference signal generation 
processing 98 are outputted to the image compression/extraction unit 76 of 
FIG. 9, and are compressed by, e.g., the JPEG format. The image data thus 
obtained after undergone compression are outputted to the external memory 
medium 51 through the microcomputer interface 73, and are stored therein. 

Since one image data is formed on the basis of four kinds of color 
signals in a manner as stated above, the reproduction characteristic thereof 
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results in that closer to seeing of the eye of the human being. 

On the other hand, when reproduction (display) of image data stored 
in the external memory medium 51 is instructed, image data stored in the 
extemal memory medium 51 is read out by the microcomputer 41. The 
image data which has been thus read out is outputted to the image 
compression/extraction unit 76 of the camera system LSI 48. At the image 
compression/extraction unit 76, compressed image data is expanded. Thus, 
image corresponding to obtained data is displayed on the image monitor 50 
through monitor interface 77. 

Then, processing (procedure) for preparing the image pick-up 
apparatus having the configuration as described above will be explained with 
reference to the flowchart shown in FIG. 11. 

At step SI, processing for determining color filter for four colors, 
which determines spectral sensitivity characteristics of the color filter 61 for 
four colors provided at the image sensor 45 shown in FIG. 5, is performed. 
At step S2, linear matrix coefficient M determination processing for 
determining matrix coefficients M, which are set at the linear matrix 
processing unit 95 shown in FIG. 10, is performed. The detail of the 
processing for determining color filter for four colors, which is executed at the 
step SI, will be described later with reference to the flowchart shown in FIG. 
,12, and the detail of the processing for determining linear matrix coefficients 
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M, which is executed at the step S2, will be described later with reference to 
the flowchart shown in FIG, 16. 

After color filter 61 for four colors is determined so that matrix 
coefficients are determined, signal processing unit 71 shown in FIG. 10 is 
prepared at step S3. Thus, processing proceeds to step S4, at which camera 
system LSI48 shown in FIG. 9 is prepared. Moreover, at step S5, the 
entirety of the image pick-up apparatus (e.g., digital camera) as shown in FIG. 
5 is prepared. At step S6, evaluation of picture quality ("color reproduction 
characteristic", "color discriminating characteristic") of the image pickup 
apparatus prepared at the step S5 is performed. Thus, processing is 
completed. 

Here, object color referred in evaluating "color reproduction 
characteristic" and "color discriminating characteristic", etc. will be explained. 
The object color is calculated by value obtained by integrating, within the 
range of visible light range (e.g., 400 to 700 nm), product of "spectral 
reflection factor of object", "spectral energy distribution of standard 
illumination", and "spectral sensitivity distribution (characteristic) of sensor 
for sensing object (color filter)". Namely, the object color is calculated by 
the formula (3). 



25 



Object color = k J . (spectral reflection factor of object) • (spectral energy distributi on of illuminati on) 

• (spectral sensitivit y distributi on of sensor for sensing object) dA * "(3) 
A : wavelength 

vis : visible light region (ordinaril y 400 nm to 700 nm) 

For example, in the case of observing a predetermined object by the 
eye, "spectral sensitivity characteristic of the sensor" of the formula (3) is 
represented by color matching function, and object colors of that object are 
represented by three stimulation values of X, Y, Z. In concrete terms, value 
of X is calculated by the formula (4-1), value of Y is calculated by the formula 
(4-2), and value of Z is calculated by the formula (4-3). It is to be noted that 
values of constant k in the formulas (4-1) to (4-3) are calculated by the 
formula (4^). 

X = f R(A) • P(A) • xiA)dA' • -(4 - 1) 

jvis 



R(A) ^spectral reflection factor of object 
z(Ay, color matching function 
k=-l/jP{A)^y(A)dA'" (4-4) 

Moreover, in the case where image of a predetermined object is taken 
thereinto by the image pick-up apparatus such as digital camera, etc., "spectral 
sensitivity characteristic of sensor" of the formula (3) is represented by the 
spectral sensitivity characteristic of the color filter, and object color of that 
object of color value (RGB value (ternary), e.g., in the case of RGB filters , 




Jvis 
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(three kinds)) is calculated. In the case where RGB filter for detecting three 
kinds of colors is provided at the image pick-up apparatus, value of R is 
calculated by the formula (5-1), value of G is calculated by the formula (5-2), 
and value of B is calculated by the formula (5-3) in practical sense. 
Moreover, value of constant k^ in the formula (5-1) is calculated by the 
formula (5-4), value of constant kg in the formula (5-2) is calculated by the 
formula (5-5), and value of constant kb in the formula (5-3) is calculated by 
the formula (5-6). 



Then, the processing for determining color filter for four colors, which 
is performed at the step SI shown in FIG. 11, will be explained with reference 
to the flowchart shown in FIG. 12. 

It is to be noted that while various methods exist as the method for 
determining color filter for four colors, explanation will be given in 
connection with the processing in which, e.g., RGB filter is caused to be base 





r(A),g(A)^b(A) : spectral sensitivity distribution of color filter 
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(one of existing (FIG. 1) G filters is caused to be Gl filter), and G2 filter for 
allowing color having high correlation with respect to color transmitted 
through the Gl filter to be transmitted therethrough is selected to supplement 
the G2 filter thus to determine color filter for four colors. 

At step S21, color target used for calculating UMG value is selected. 
For example, at step S21, there is selected color target in which many color 
patches representing existing colors are included and many color patches 
where importance is attached to memory color of the human being (skin color, 
green of plant, blue of sky, etc.) are included. As color target, there is, e.g., 
IT8.7, Macbeth Color Checker, GretargMacbeth DigitalCamera Color 
Checker, CIE, Color Bar, etc. 

Moreover, according to the object, color patch which can serve as 
standard may be prepared from data such as SOCS (Standard Object Color 
Spectra Database), etc. to use the color patch thus prepared. It is to be noted 
that the detail of the SOCS is disclosed in Uohji Tajima, "Statistical Color 
Reproduction Evaluation by Standard Object Color Spectra Database 
(SOCS)", Color forum JAPAN 99\ Explanation will be given below in 
connection with the case where Machbeth Color Checker is selected as color 
target. 

At step S22, spectral sensitivity characteristic of G2 filter is 
determined. As the, spectral sensitivity characteristic, there may be used 
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spectral sensitivity characteristic which can be prepared from existing material, 
or there may be used spectral sensitivity characteristic obtained by assuming 
virtual curve C(X) by cubic spline curve (cubic spline function) as shown in 
FIG. 13 to change peak value Xq of virtual curve C(X), value w (value 
obtained by dividing sum of wi and W2 by 2) and value Aw (value obtained by 
dividing value obtained by subtracting W2 from Wi by 2) within the range 
shown. It is to be noted that values of w and Aw are set to values based on 
value of half-value width. As a method of changing Aq, w. Aw, those values 
are assumed to be changed at pitch of e.g., 5nm. The virtual curves C(X) are 
represented by the following formulas (6-1) to (6-5) within respective ranges. 



^,,,^^1^ 3wl (w^ - |;i - ;io|) + (W2 -\A-Ao\y'3(w,-\A-Ao\y 

C(A) 

owl 

_ + 3wf (w, -\A'A^\)^ 3w, (w, -|.a-^o|)^-3(w, -[^j-^j)^ 
C{A) : 

A - w, ^ >a - ^ ^ 0- • -(6 - 2) 
C(;i) = ^'"-"JV^^'^' A w,^;i-^o^2w,---(6-3) 

C(;i)=(?l!L^tL:Ml ^ -2>v.^;i-A„^-w,.-.(6-4) 
6>v, 

C(A) = 0 A except for the above range * • -(6 - 5) 



It should be noted that although only filter G2 is supplemented in this 
example, only R filter and B filter of filter (R, G, G, B) shown in FIG. 1 may 
be used to define the remaining two Gl, G2 filters as virtual curve of the 
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above^entioned formulas (6-1) to (6-5) in the vicinity of green color. 
Moreover, similarly, only R and G, and only G and B may be used from the 
filter shown in FIG. 1. Further, among filters for four colors, filters for three 
colors may be defined as virtual curve, and filters for four colors may be 
defined all as virtual curve. 

At step S23, filter to be supplemented (G2 filter) and existing filters 
(R filter, Gl filter, B filter) are combined so that the minimum unit (set) of 
color filters for four colors is prepared. Moreover, at step S24, UMG is used 
as filter evaluation coefficients with respect to color filter for four colors 
prepared at the step S23. Thus, UMG value is calculated. 

As explained with reference to FIG. 8, in the case where UMG is used, 
evaluation can be performed at a time with respect to respective color filters 
for four colors. Moreover, not only evaluation is performed by taking into 
consideration spectral reflection factor of object, but also evaluation is made 
by taking into consideration reduction characteristic of noise. Since high 
evaluation is indicated with respect to filters having suitable overlap in 
spectral sensitivity characteristics of respective filters in the evaluation using 
UMG, it is possible to suppress that high evaluation is indicated with respect 
to, e.g., filter having the characteristic in which the characteristic of R and the 
characteristic of G overlap with each other over a broad wavelength band 
(filter in which noise is amplified when respective color signals are separated). 
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FIG. 14 is a view showing example of UMG value calculated at the 
color filter for three colors. For example, in the filter having the characteristic 
as shown in FIG. 14(A) in which characteristics of RGB do not overlap with 
each other, UMG value of "0.7942" is calculated. In the filter having the 
characteristic as shown in FIG. 14(B) in which characteristic of R and 
characteristic of G overlap with each other over a broad wavelength band, 
UMG value of "0.8211" is calculated. Moreover, in the filter having the 
characteristic as shown in FIG. 14(C) in which respective characteristics of 
RGB suitably overlap with each other, UMG value of "0.8879" is calculated. 
Namely, the highest evaluation is indicated with respect to the filter having the 
characteristic as shown in FIG. 14(C) in which respective characteristics of 
RGB suitably overlap with each other. This similarly applies also to color 
filter for four colors. In this example, curve L31 shown in FIG. 14(A), curve 
L41 shown in FIG. 14(B) and curve L51 shown in FIG. 14(C) represent 
spectral sensitivity of R, curve L32 shown in FIG. 14(A), curve L42 shown in 
FIG. 14(B) and curve L52 shown in FIG. 14(C) represent spectral sensitivity 
of G, and curve L33 shown in FIG. 14(A), curve L43 shown in FIG. 14(B) 
and curve L53 shown in FIG. 14(C) represents spectral sensitivity of B. 

At step S25, whether or not UMG value calculated at step S24 is 
"0.95" which is predetermined threshold value or more is judged. In the case 
where it is judged that the UMG value is less than "0.95", processing proceeds 
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to Step S26 so that the prepared color filter for four colors is rejected (is not 
used). At step S26, in the case where color filter for four colors is rejected, the 
processing is completed thereafter (processing of the step S2 shown in FIG. 11 
and those subsequent thereto are not executed). 

On the other hand, at the step S25, in the case where it is judged that 
UMG value calculated at the step S24 is "0.95" or more, that color filter for 
four colors is caused to be candidate filter used in the digital camera at step 
S27. 

Whether or not the color filter for four colors caused to be candidate 
filter at the step S27 can be realized by existing material/dye is judged at step 
S28. In the case where it is difficult to acquire material/dye, etc., it is judged 
that such color filter cannot be realized. Processing proceeds to step S26. As a 
result, the color filter for four colors is rejected. 

On the other hand, in the case where it is judged that material/dye, etc. 
can be acquired so that such color filter can be realized, processing proceeds 
to step S29. Thus, the color filter for four colors thus prepared is determined 
as filter used in the digital camera. Thereafter, processing of step S2 shown in 
FIG. 1 1 and those subsequent thereto are executed. 

FIG. 15 is a view showing an example of spectral sensitivity 
characteristic of color filter for four colors which has been determined at step 
S29. 
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In FIG. 15, curve L61 represents spectral sensitivity of R, and curve 
L62 represents spectral sensitivity of GL Moreover, curve L63 represents 
spectral sensitivity of G2, and curve L64 represents spectral sensitivity of B, 
As shown in FIG. 15, the spectral sensitivity curve of G2 (curve L63) has high 
correlation with respect to the spectral sensitivities curve of Gl (curve L62). 
Further, spectral sensitivity of R, spectral sensitivity of G (Gl, G2) and 
spectral sensitivity of B overlap with each other within suitable range. 

By utilizing the color filter for four colors determined in a manner as 
stated above, it becomes possible to improve particularly "discriminating 
characteristic of color" of "color reproduction characteristic". 

It is to be noted that it is preferable that filter having high correlation 
with respect to G filter of the existing RGB filter is caused to be filter to be 
supplemented (G2 filter) in a manner as stated above from viewpoint of 
utilization efficiency of light. In this case, it is desirable that peak value of the 
spectral sensitivity curve of filter to be supplemented experimentally exists 
within the range from 495 to 535 nm (in the vicinity of peak value of the 
spectral sensitivity curve of the existing G filter). 

Moreover, in the case where filter having high correlation with respect 
to the existing G filter, since either one of two G filter constituting the 
minimum unit (R, G, G, B) shown in FIG. 1 is only caused to be filter of color 
to be supplemented to thereby have ability to prepare color filter four colors. 
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there is no necessity to greatly change production process step. 

In the case where color filter for four colors is prepared in a manner as 
stated above and the color filter thus prepared is provided at the digital camera, 
since four kinds of color signals are delivered from the signal generation 
processing unit 94 at the signal processing unit 71 shown in FIG. 10, 
transform processing for generating signals of three colors (R, G, B) from 
signals of four colors (R, Gl, G2, B) is performed at the linear matrix 
processing unit 95. Since this transform processing is matrix processing with 
respect to luminous linear (luminance value can be represented by linear 
transform processing) input signal value, transform processing performed at 
the linear matrix processing unit 95 will be called linear matrix processing 
hereinafter as occasion demands. 

Then, the linear matrix coefficient M determination processing, which 
is executed at the step S2 shown in FIG. 11, will be explained with reference 
to the flowchart shown in FIG. 16. It is to be noted that color target used in the 
linear matrix coefficient M determination processing is caused to be Macbeth 
Color Checker, and the color filter for four colors used is caused to have 
spectral sensitivity characteristic shown in FIG. 15. 

At step S41, e.g., general day light D65 (illumination light L(X)) used 
as standard light source in CIE (Commision IntemationalederEclairange) is 
selected as illumination light. It is to be noted that the illumination light may 



34 



be changed into illumination light, etc. of the environment where it is 
expected that the image processing apparatus is frequently used. Further, in 
the case where plural illumination environments assumed exist, it is 
conceivable to prepare plural linear matrices. The case where the day light 
D65 is selected as illumination light will now be explained. ' 

At step S42, reference values Xr, Yr, Zr are calculated. In concrete 
terms, the reference value Xr is calculated by the formula (7-1), the reference 
value Yr is calculated by the formula (7-2), and the reference value Zr is 
calculated by the formula (7-3). 




R(A) ^spectol reflection factor of object 
y (/I), z(^) : color matching function 



Moreover, constant k is calculated by the formula (8). 




For example, in the case where color target is Macbeth Color Checker, 
reference values corresponding to 24 colors are calculated. 

Then, at step S43, output values Rf, Glf, G2f, Bf of color filters for 



four colors are calculated. In concrete terms, Rf is calculated by the formula 
(9-1), Glf is calculated by the formula (9-2), G2f is calculated by the formula 
(9-3), and Bf is calculated by the formula (9-4). 



Glf=k^^l R(A)^LiA)*gKA)dA'-'i9-2) 
^^f ' *.2jl^(^)*i(>^)^»g2(/l)rf>a- • -(9-3) 
Bf=kA R{A) • L{A) • HA)dA • • -(9 - 4) 

^i^)^8K^Xg'2>iA),b(A) : spectral sensitivity distribution of color filter 

Moreover, constant is calculated by the formula (10-1), constant Kgi 
is calculated by the formula (10-2), constant Kg2 is calculated by the formula 
(10-3), and constant kb is calculated by the formula (10-4). 

it, =l/f L(A)^r(A)dA-' (10 -l) 

Jvis 

fc„ = 1/ f LiA) • YKA)dA (10-2) 

* Jvis 

fe,2=l/f L(A)»'i2(A)dA'-' (10-3) 

* Jvis 

k.=l/{ LiA)»biA)dA"' (10-4) 

Jvis 

For example, in the case where color target is Macbeth Color Checker, 
output values Rf, Glf, G2f, Bf corresponding to 24 colors are calculated- 

At step S44, matrix for performing transform processing to 
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approximate filter output value calculated at the step S43 into reference values 
(XYZref) calculated at the step S42 is calculated by, e.g., error least square 
method in the XYZ color space. 

For example, in the case where matrix of 3 x 4 to be calculated is 
assumed to be A represented by the formula (11), matrix transform (XYZexp) 
is represented by the following formula (12). 



A = 



aO al ah a4 
a4 a5 a6 al 
aS a9 alO all 



•(11) 





X 




"aO 


a2 


a3 


a4 


XYZexp = 


Y 




a4 


a5 


a6 


al 




z 




aS 


a9 


alO 


all 



G2, 



•(12) 



Moreover, square (E2) of error of matrix transform (formula (12)) 
with respect to the reference value is represented by the following formula 
(13). On the basis of this value, matrix A which minimizes error of matrix 
transform with respect to reference value is calculated. 



= \XYZref - X^Zexpf • • '(13) 



Further, color space used in the error least square method may be 
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changed into color space except for XYZ color space. For example, there 
may be performed transform processing into Lab, Luv, Lch color spaces even 
with respect to sensory perception of the human being (perceptive uniform 
color space) thereafter to perform similar operation, thereby making it 
possible to calculate linear matrix which permits reproduction of color less in 
perceptive error. It is to be noted that since these values of color space are 
calculated from XYZ values by non-linear transform processing, non-linear 
calculation algorithm is used also in the error least square method. 

By operation as described above, matrix coefficients represented by 
the formula (14) are calculated as matrix coefficients with respect to filter 
having spectral sensitivity characteristic shown in FIG, 15, for example. 



0.476 0.905 0.261 -0.691 
0.2 1.154 -0.061 -0.292 
-0.004 0.148 0.148 -0.481 



•(14) 



At step S45, linear matrix is determined. For example, in the case 
where final RGB image data to be prepared is assumed to be represented by 
the following formula (15), linear matrix (LinearM) is calculated in a manner 
as described below. 



RGBout = [Ro, Go, Bq] ^ (15) 
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Namely, in the case where illumination light is D65, transform 
expression to transform sRGB color space into XYZ color space is 
represented by the formula (16) including ITU-R709.BT matrix, and the 
formula (17) is calculated by inverse matrix of the ITU-R709.BT matrix. 



X 
Y 
Z 

K 
G 

B 



sRGB 



'sRCB 



sRGB 



'sRCB 



sRCB . 



0.4124 0.3576 0.1805 R 
0.2126 0.7152 0.0722 • 
0.0193 0.1192 0.9505 B 

3.2406 -1.5372 -0.4986 
-0.9689 1.8758 0.0415 
0.0557 -0.204 1.057 



■ (16) 

X' 
Y 
Z 



•(17) 



By the matrix transform expression of the formula (12), and inverse 
matrix of ITU-R709.BT matrix of the formulas (15) and (17), the formula (18) 
is calculated. In the right side of the formula (18), there are included inverse 
matrix of ITU-41709.BT matrix £ind linear matrix as value multiplied by the 
above-described matrix A. 







Go 









3.2406 -1.5372 -0.4986 
-0.9689 1.8758 0.0415 
0.0557 - 0.204 1.057 



aO a2 a3 a4 
a4 a5 a6 al 
a8 a9 alO all 



' -(18) 
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Namely, linear matrix (LinearM) of 3 ^ 4 is represented by the 
formula (19-1). Linear matrix with respect to color filter for four colors 
having spectral distribution characteristic shown in FIG. 15 in which, e.g., 
matrix coefficients of the formula (14) are used is represented by the formula 
(19-2). 



LinearM = 



la 12 13 14 

14 15 16 11 

15 19 no 111 
3.2406 -1.5372 



LinearM = 



-0.9689 
0.0557 
1.238 

-0.087 



-0.4986 
0.0415 
1.057 



1.8758 
-0.204 

1.084 0.228 
1.295 -0.309 



aO 
a4 
aS 



a2 
a5 
a9 



-1.55 
0.101 



a3 
a6 
alO 



a4 
al 
all 



••(19-1) 



• • ^(19 - 2) 



-0.018 -0.029 1.535 -0.485 



The linear matrix calculated in a manner as described above is 
delivered to linear matrix processing unit 95 shown in FIG. 10. Thus, since 
it is possible to perform matrix processing with respect to signals (R, Gl, G2, 
B) in which luminance can be represented by linear transform, it is possible to 
reproduce color having higher fidelity from a viewpoint of color engineering 
as compared to the case where matrix processing is performed with respect to 
signal obtained after gamma processing is implemented as in the case of 
processing at the signal processing unit 11 shown in FIG. 2. 
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Then, evaluation performed at step S6 shown in FIG. 11 will be 
explained. 

In the case where comparison between color reproduction 

characteristic of the image pick-up apparatus provided with color filter for 

♦ 

four colors having, e.g., spectral sensitivity characteristic shown in FIG. 15 
which have been prepared in a manner as stated above and color reproduction 
characteristic of the image processing unit provided with color filter for three 
colors shown in FIG. 1 is performed, difference as described below appears. 

For example, color differences at Lab color space between output 
values when image of Macbeth chart is picked up by two kinds of image input 
apparatuses (image pickup apparatus provided with color filter for four colors 
and image pick-up apparatus provided with color filter for three colors) and 
reference values are respectively calculated by the following formula (20). 

A£ = V(A 'hf^ («i - «2)' + (^1 -^2)'- • '(20) 

Li - L2 indicates lightness difference between two samples, and ai - a2, 
bi - bi indicate component difference of hue/saturation of two samples. 

FIG. 17 is a view showing calculation result by the formula (20). As 
shown in FIG. 17, in the case of the image pick-^p apparatus provided with 
color filter for three colors, color difference is "3.32", whereas in the case of 
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the image pick-up apparatus provided with color filter for four colors, color 
difference is "1.39". "Way of seeing of color'' at the image pick-up 
apparatus provided with color filter for four colors is more excellent (color 
difference is small). 

In FIG. 18, R value of object Rl is set to "49.4", G value thereof is set 
to "64.1" and B value thereof is set to "149.5", and R value of object R2 is set 
to "66.0", G value thereof is set to "63.7" and B value thereof is set to "155.6". 
Accordingly, in the color filter for four colors, RGB values of the object Rl 
and those of the object R2 become values different from each other. 
Similarly to the case where object is seen by the eye, colors of respective 
objects are discriminated. Namely, filter capable of discriminating between 
four kinds of colors is provided so that "discriminating characteristic of color" 
is improved. 

While the color filter 61 for four colors is constituted in the above 
example by arrangement such that B filters are provided at the left and right 
sides of Gl filter, and R filters are provided at left and right sides of the G2 
filter as shown in FIG. 6, such color filter may be constituted by arrangement 
as shown in FIG. 19. In the color filter 61 for four colors shown in FIG. 19, 
R filters are provided at left and right sides of the Gl filter, and B filters are 
provided at left and right sides of the G2 filter. Also by constituting the 
color filter 61 for four colors in this way, it becomes possible to improve 
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"discriminating characteristic of color", "reproduction characteristic of color" 
and "reduction characteristic of noise" similarly to the color filter shown in 
FIG. 6. 

Meanwhile, in the case where linear matrix coefficients M are 
determined in such a manner to minimize color difference (AE value), if 
spectral sensitivities of color filters formed at the front stage portion of the 
image sensor 45 overlap with each other as shown in FIG. 4, differences 
between linear matrix coefficients M become large as indicated by the formula 
(21), 



'rXA)' 




' 6.56 


-5.54 


1.18 ■ 


"r(A)' 








-2.01 


3.12 


-0.16 


• 


- (21) 






0.12 


-0.28 


1.07 


HA) 





Since very small noise component is included in an output signal of 
the image pick-^ip device, when such linear matrix coefficients M are used to 
perform color separation processing, very small noise component is also 
amplified. Accordingly, there takes place the necessity in which noise 
reduction characteristic is taken into consideration rather than color 
reproduction characteristic so that differences between linear matrix 
coefficients M do not large. However, there are instances where when image 
of object is actually picked up by such image pick-up apparatus, there is 
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employed a method in which linear matrix coefficients M are determined 
under the condition where importance is attached to noise reduction 
characteristic rather than color reproduction characteristic by scene and/or 
environment to be imaged to adaptively perform linear matrix processing so 
that improvement in picture quality can be realized. Conversely, there are 
instances where there is employed a method in which importance is attached 
to color reproduction characteristic rather than noise reduction characteristic 
to determine linear matrix coefficients M to adaptively perform linear matrix 
processing so that improvement in picture quality can be realized. In 
addition, since use purpose of the image pick-up apparatus is different every 
user, there are instances where user desires to arbitrarily perform 
determination of linear matrix coefficients M. 

In view of the above, in the image pick-up apparatus according to the 
invention of this Application, in order to solve problems as described above, 
linear matrix coefficients M are determined in accordance with the flowchart 
shown in FIG. 20. 

First, determination of chart and illumination light which are used is 
performed (step S50). Then, definition of color reproduction characteristic 
index AE(M) is performed (step S51). Then, definition of noise reduction 
index aN(M) is performed (step S52). Then, definition of EEV(M) (Error 
Evaluation Value) is performed (step S53). Further, linear matrix 
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coefficients M are determined on the basis of the EEV (M) (step S54). It is 
to be noted that, at the step S54, coefficients of EEV(M) are adaptively 
changed depending upon image pick-up condition, etc. to determine 
corresponding linear matrix coefficients M. The detail of respective steps 
will be explained below. 

Determination of chart and illumination light which are used (step 
S50) will be explained. In order to determine linear matrix coefficients M, it 
is necessary to determine color chart and light source for illuminating the 
color chart. As color chart, there are conceivable various reflection charts or 
transmission charts consisting of color patches having plural uniform color 
planes such as Macbeth Color Checker, Digital Camera Color Checker, ITS .7, 
etc. As illumination light, there is conceivable illumination light (e.g., D55 
light source, etc.) having spectral sensitivity close to light of the environment 
where the image pick-up apparatus is frequently used. It is to be noted that 
since it is conceivable that the mage pick-^p apparatus is used under various 
light sources depending upon use purpose of user, illumination light is not 
limited to only light of the environment where the image pick-up apparatus is 
frequently used as illumination light. 

Then, definition of color reproduction characteristic index AE(M) 
(step S51) will be explained. The color reproduction characteristic is defined 
by difference between target color and color (hereinafter referred to as output 
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color) that signal value in which linear matrix processing is performed at 
linear matrix processing unit 95 of the image pick-up apparatus indicates. It is 
to be noted that while RGB values, YCbCr values or XYZ values, etc, are 
variously conceivable as value of color, a method of performing definition 
processing by using values (L*a*b* value, L*u*v* value, etc.) of color space 
where sensory perception is uniform with respect to seeing of the eye of the 
human being makes it possible to more precisely indicate difference of color. 
For example, when target color of the k-th color patch in the color chart is 
assumed to be Labref_k (L*ref_k, a *ref_k, b*ref_k), and output color of the image 
pick-up apparatus is assumed to be L*a*b*shot_k (L*shot_k. a*shot_k5 b*shoek)5 
color difference AEk of this patch is represented by the formula (22). 

AE, -^{L^ref^k -L^sHot.k)^ ^ici^ref^k * shot ^ * ref * sHot ^kf i^^) 



Moreover, as color reproduction characteristic index AE(M), there are 
conceivable average AE value of respective patches of the color chart, and/or 
value in which weighting is performed with respect to respective patches so 
that importance is attached to color reproduction characteristic of a specific 
color, etc. 
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I Total Patch Num 

AE ' f AE, (23) 

TotalPatchNum ^J, 

In the above formula, Wk indicates weighting coefficients with respect 
to respective patches, and TotalPatchNum indicates the total number of color 
patches. 

Moreover, since linear matrix processing is implemented to output 
color of the image pickup apparatus in practice, L*a*b*shot_k indicates 
function value of linear matrix coefficients M. Accordingly, AEk and AE both 
result in function value of M. 



I TotalPatcNum 

TotalPatcNum , * * 



TotalPatcNum 



TotalPatcNum 



(24) 



Then, definition of noise reduction characteristic index aN(M) (step 
S52) will be explained. The noise reduction characteristic index aN(M) is 
defined by standard deviation of signal values in which linear matrix 
processing is performed at linear matrix processing unit 95 of the image 
pick-up apparatus. As signal value, there are conceivable RGB values, YCbCr 
values or XYZ values, etc. In this case, a method of defining signal value by 
using values of color space where sensory perception is uniform with respect 
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to seeing of the eye of the human being (L*a*b*value, L*u*v*value), etc. 
makes it possible to obtain noise value aNk having higher correlation with 
respect to noise feeling that the human being feels. At this time, noise value 
results in standard deviation of respective components of color space in 
correspondence with color space of signal value. For example, in the case of 
RGB space, noise value results in aR, oG, oB. In the case of XYZ space, 
noise value results in aX, oY, aZ. In the definition of the noise reduction 
characteristic index aN(M), these noise values are used to determine single 
noise index. For example, in the case where image of a certain color patch is 
picked up under illumination light, noise value oN^ of L*a*b* space is 
defined from values of lightness noise aL*k and color noises aa*k5 <7b*k on 
the premise that lightness and color noise are taken into consideration, as 
indicated by, e.g., the formula (25). 



^^.I^IJf^lJ^I ...(25) 

In this case, wL*k, wa*k5 wb*k indicate weighting coefficients with 
respect to respective standard deviation values, and are suitably set by the 
correlation with respect to noise feeling that the eye of the human being feels. 
It is to be noted that noise value using variance value of other color space, etc. 
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may be variously conceivable as noise value oNk. As noise reduction 
characteristic index aN(M), there are conceivable average aN value of 
respective patches of color chart, and/or value in which weighting is 
performed with respect to respective patches so that importance is attached to 
noise reduction characteristic of a specific color, etc. 



In practice, since linear matrix processing is implemented to signal 
value of the image pick-up apparatus, oN^ and oN result in function values of 
the linear matrix coefficients M. 



Then, definition of EEV (M) (step S53) will be explained. By the 
definition of the above-described steps S51 and S52, EEV (Error Evaluation 
Value) (M) in which two values of color reproduction characteristic index 
AE(M) and noise reduction characteristic index aN(M) which are fiinction 
values of the linear matrix coefficients M are taken into consideration is 




1 



TotalPatchNum 



TotalPatchNum 



jw, cN,(M) -(27) 
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defined in a manner as indicated by the formula (28). 

EEViM) = iy{wc h{AE(M))] + k{wn'i{oNiM)))] (28) 

In the above formula, h, I, j, k, 1 indicate function, wc indicates 
weighting coefficient with respect to color difference, and wn indicates 
weighting coefficient with respect to noise value. By changing wc and wn to 
determine linear matrix coefficients M so that EEV (M) becomes minimum, 
determination of linear matrix coefficients M in which both color reproduction 
characteristic and noise reproduction characteristic are taken into 
consideration can be made. It is to be noted that it is sufficient that in the case 
where importance is attached to color reproduction characteristic, weighting is 
set so that wc > wn, and in the case where importance is attached to noise 
reduction characteristic, weighting is set so that wc < wn. 

Then, means for determining linear matrix coefficients M (step S54) 
will be explained. The error least square method is applied to the EEV (M) 
defined by the step S53 to determine linear matrix coefficients M. In this case, 
at step S54, wc and wn are suitably determined to use, e.g., Newton method. 
Steepest Descent method or Conjugate Gradient method, etc. as recursive 
algorithm and to apply error least square method, and thus to determine linear 
matrix coefficients M. 
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Moreover, at step S54, weighting coefficient wc with respect to color 
difference and weighting coefficient wn with respect to noise value of EEV 
(M) defined at the step S53 are adaptively changed by environment and/or 
condition, etc. when image of object is picked up by the image pick-up 
apparatus to determine linear matrix coefficients M by the error least square 
method. The state of change of noise value reduction characteristic index 
aN(M) when color reproduction characteristic index AE(M) is changed is 
shown in FIG. 21. As shown in FIG. 21, also in one image pick-up apparatus, 
trade-off of color reproduction characteristic index AE(M) and noise reduction 
characteristic index aN(M) exists by linear matrix coefficients M. By using 
this result, linear matrix coefficients M are adaptively determined in 
accordance with various image pickup environments and conditions, etc. 
Moreover, several sets of linear matrix coefficients M may be prepared in 
advance to allow user to select linear matrix coefficients M as occasion 
demands to adjust color reproduction characteristic index AE(M) and noise 
reduction characteristic index aN(M). 

Here, a practical example in which in the case where the image 
pick-up apparatus includes CCD image pick-up device comprised of color 
filters for four colors having the characteristic as shown in FIG. 22, linear 
matrix coefficients M are determined in accordance with the above-described 
steps S50 to S54 will be described. . 



51 



First, chart and illumination light which are used are determined (step 
S50). As color chart, Macbeth Color Checker (including color patches of 24 
colors) is used. As illumination light, D55 light source (standard day light of 
5500k defined by CIE) is used. It is to be noted that it is assumed that 
respective spectral data are measured by using, e.g., spectral radiation 
luminance meter. 

Then, definition of color reproduction characteristic index AE(M) is 
performed (step S51). Target color is caused to correspond to seeing of the eye 
of the human being, and color difference AE in Lab space is caused to be 
index. In general, object color is defined, as indicated by the formula (29), by 
value obtained by integrating product of "spectral reflection factor of object", 
"spectral energy distribution of illumination" and "spectral sensitivity 
distribution of sensor for sensing object" within the range of visible light 
region vis (ordinarily 400 nm to 700 nm). 



Objectcolor = J (spectralreflectionfactorof object)- (spectralluminanceof illuminatfan) 
• (spectralsensitiviv of sensorfor sensingobject) 



Further, when color matching function representing spectral 
sensitivity of the eye of the human being is used, target colors XYZref_k (Xref_k, 



- (29) 
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Yref_k, Zref_k) of the k-th patch of the color chart can be represented by the 
fomula (30) by using the fomula (29). 

In theabovefoinula,/;;^ (A) : Spectralreflectionfactorof thek -th colorpatch witlin Macbethchart 
LW^ Uluminatbn light DS5 spectralradiationluminance 

x,y,z: color matching function 

Moreover, ordinarily, colors of the XYZ space are transformed into 
colors of L*a*b* space by using the formula (31). 



L* = 116 
a* = 500 
d* = 200- 



^y/F„)"' -16 
i(y/y„)"'-(z/z„)"^J 



(31) 



In the above formula, (Xn, Yn, Zn) indicates XYZ values of complete 
diffused reflection surface (white point). 

Further, target color XYZref_k is transformed into L*a*b*ref_k(L*ref_k5 
a*ref_k, b*ref_k) by usiug the formula (31). 

In addition, raw data RGBXraw.k (Rraw.k, Graw^k, Braw.k, Xraw^k) which 
are signal values outputted from the CCD image pick-up device are 
represented by the formula (32) by using the formula (29). 
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raw_^k 



= mg 



Bra^t = rnb 



X. 



= mx 



^^R,(A)L(A)xiA)dA 



(32) 



In theabovefonnula,r,g,i>,x : spectralsensitivity distribution of CCD color filter of camera 



^LiA)giA)dA 
,L(A)biA)dA 
L(A)xiA)dA 



mr 


= 1/ 


mg 


= 1/ 


mb 


= 1/ 


mx 


= 1/ 



Since the image pick^ip apparatus performs linear matrix processing 
of raw data RGBXraw_k(Rraw_k, Graw_k5 Braw_k5 Xraw_k) by using linear matrix 
coefficients M (mO to mil) at the linar matrix processing unit 95, image 
pick-up data after undergone linear matrix processing is represented by the 
formula (33). 



R 



^camk 



cam_k 



= M 



raw k 



B 



cam _ k 



B 



raw _k 



raw k 



mO ml m2 m3 
m4 m5 m6 ml 
mS m9 mlO mil 



R 



B 



raw^ k 



(33) 
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Further, the RGBcam.k is transformed into XYZ value. It is to be noted 
that 709 system matrix M709 generally used is used as shown in the formula 
(34) for the transform processing. 



'x' 




R 




Y 




G 




Z 




B 





0.4124 0.3576 
-0.9689 1.8758 
0.0557 - 0.204 



0.1805 




R 




0.0415 




G 


••- (34) 


1.057 




B 





Then, Xcam.k, Ycam_k and Zcam_k fTc determined by using the formula 



(34). 







0.4124 


Y 

cam_k 




-0.9689 


^cam_k 




0.0557 



-0.204 1.057 



R 



cam _ k 



*cam_k 



B 



cam_k 



(35) 



Moreover, transform processing into L*a*b* value 
(L*a*b*cam_k(L*cam_k, a*cam_k, b*cam_k) IS performed by using the formula (31) 
to define color difference AEk of the patch of the color chart by the 
formula (36). 



AE, -^{L*ref_k -i^\am_kf^(^\ef.k W*)' "^W*)' -(36) 



It is to be noted that since value of L*a*b*cain k is used, AEk can be 
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represented as AEk(M) because it is also function value of the linear matrix 
coefficients M. The color reproduction characteristic index AE(M) is defined 
as average value of color differences of respective color patches as shown in 
the formula (37). 



In the formulSiyTotalPatchNum = 24 : Total number of color patches 

Then, definition of noise reduction characteristic index aN(M) is 
performed (step S52). Here, noise reduction characteristic index aN(M) is 
defined on the basis of aL(M) component included in signal value after 
undergone linear matrix processing by the linear matrix processing unit 95 of 
the image pick-up apparatus. 

In general, noise Noise^j-aw included in signal CV_ccd that the CCD 
image pick-up device itself outputs is defined by the formula (38). 



Noise = J ShotNoiseCoef • CV + DarkNoise 




ToalPatchNum 



••■(37) 



= V ShotNoise + DarkNoise 



• • -(38) 



It is to be noted that ShotNoiseCoef and DarkNoise are values 
determined by the device characteristic of the CCD image pick-up device. 
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DarkNoise represents noise component which is not dependent upon signal 
value (Fixed Pattern Noise, etc.), and ShotNoise represents noise component 
dependent upon signal value (Sensor Dark Noise, Photon Shot Noise, etc.). 

When the formula (31) is used, noise components included in raw data 
of the k-th color patch of the image pick-up apparatus to be evaluated are 
defined by the formula (39). 





J = ^ShotNoiseCoef 




+ DarkNoise 




^ = yj ShotNoiseCoef 


G 

raw 


+ DarkNoise 




1^ = ^ShotNoiseCoef • 


^raw_ 


^ + DarkNoise 




J = ^ShotNoiseCoef ■ 


X 

raw. 


^ + DarkNoise 



(39) 



Moreover, in the literature (RD. Burns and R.S. Berns, "Error 
Propagation Analysis in Color Measurament and Imaging", Color Research 
and Application, 1997), noise propagation theory as explained below is 
described. 

When a predetermined input signal X' = [xi, X2, , xj is assumed to 
be linearly transformed into Y' = [yi, y2, ym] by (m x n) matrix A, linearly 
transformed signal is represented by the formula (40). 



= A X' - (40) 
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When variance-covariance matrix Lx of input signal X' is assumed to 
be represented by the formula (41), diagonal component results in Noise 
variance value of input signal. 



5:. = 



cr,2 

^22 



'In 



(41) 



If there is no correlation between input signal values mutually, 
covariance component (i.e., non-diagonal component) in the matrix 
component becomes equal to zero. At this time, variance-covariance matrix 
Ly of output signal Y is defined by the formula (42). 

E, = ^ L,^' " (42) 

It is to be noted that the formula (42) results in propagation theoretic 
formula of Noise variance value between color spaces which can be 
transformed by linear transform. 

Moreover, in order to transform signal RGBcam_k after undergone 
linear matrix processing by the linear matrix processing unit 95 of the image 
pick-up apparatus into L*a*b*cam_k5 it is necessary to perform transform 
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processing from RGB space to XYZ space (hereinafter referred to as RGB — > 
XYZ transform processing) thereafter to perform transform processing from 
XYZ space to L*a*b*space (hereinafter referred to as XYZ — > L*a*b* 
transform processing). In the RGB —> XYZ transform processing, it is 
possible to perform hnear transform by using 709 system M709 shown in the 
formula (34), whereas in the XYZ L*a*b* transform processing, it is 
necessary to perform non-linear transform in a manner as shown in the 
formula (31). However, in the XYZ -> L*a*b* transform processing, linear 
approximation can be made by using Jacobian matrix JL*a*b*_k because noise 
quantity is very small. Thus, XYZ —> L*a*b* transform processing can be 
performed by linear transform in a manner similar to RGB — > XYZ transform 
processing. 

When value obtained by transforming signal value after linear matrix 
into XYZ value is expressed as XYZcam^k (Xcam_kj Ycam.k? Zcam.k)^ this value 
can be represented by the formula (43). 



JL*a*b*_k " JoE>(XYZcam_k) (43) 



In the above formula, Jo is expressed as the formula (44). 
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•^0 = 



0 116 0 " 
500 -500 0 •••(44) 

0 200 -200 
Moreover, when v{a,b) is expressed as below, 

a 



v(a,b) = ^ 



1 

— a 
3 



2 I 

s-b'i -> 0.008856 
b 



7.787 -fc"' 



- < 0.008856 
b 

the formula (45) is provided. 

^v(X_ „XJ 0 

0 0 



DiXYZ^^_,) = 



0 
0 



(45) 



In the above formula, XYZ value of complete diffused reflection surface (white point) is 
set to XYZn (Xn, Yn, Zn). 



Accordingly, approximate matrix Mtotai.k for linearly transforming raw 
data outputted from the CCD image pick^ip device into Lab value is 
expressed as the formula (46) 

^ total _k = 'fL*a*b*_k 'M-j^'M (46) 



When the matrix of the formula (46) and the formula of noise 
propagation theory shown in formula (42) are applied, lightness noise aLk at 
the k-4h color patch can be calculated by the formula (47). 
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0 0 0 

NoiseRawG_k 0 ^ Af ' 

0 NoiseRawBk 0 

0 0 NoiseRawXk 

(47) 

Accordingly, it is possible to derive the formula (48) from the formula 

(47). 

aL-,-4a'L\ •(48) 

It is to be noted that since the formula (48) is function of linear matrix 
coefficients M, this formula can be represented by aL*k (M). Since noise 
reduction characteristic index aN(M) is average value of respective lightness 
noises of color patch, aN(M) can be defined by the formula (49). 

24 

aN(M) = (l/24)- J^?l*jt iM) (49) 

Jk-l 

Then, EEV(M) in which color reproduction characteristic index 
AE(M) and noise reduction characteristic index aN(M) which are defined as 
described above ar^ taken into consideration is defined by the formula (50) 



•-RGBraw_k 



total_k 



total _k 

NoiseRawRk 
0 
0 
0 
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(step S53). 



EEV(M) = ^(wc • AE(M)y + (wn • oN{M)f • • -(50) 
In the above formula, 

wc : weighting coefficient with respect to color reproduction characteristic 
wn : weighting coefficient with respect to noise reduction characteristic. 



Then, the formula (50) is solved by the error least square method to 
determine linear matrix coefficients M. When, e.g., wc is set to 1, and wn is 
set to 2, EEV(M) is represented by the formula (51). 

EEV(M) = Va£(M)2 + (2 • oN(M)f (51) 



The formula (51) is solved by the error least square method to 
determine linear matrix coefficients M by the formula (52) (step S54). 



1.57 -0.43 -0.01 -0.12 
M = -0.14 1.25 -0.37 0.26 
-0.01 -0.27 1.68 -0.40 



(52) 



On the other hand, color difference matrix in which wc is set to 1 and 
wn is set to 0, i.e., importance is attached (attention is paid) to only color 
reproduction characteristic is represented by the formula (53) 
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1.48 0.56 0.35 -1.39 
M = -0.22 2.19 -0.01 -0.96 
-0.06 0.27 1.93 -1.14 



(53) 



When comparison between the formula (52) and the formula (53) is 
performed, it can be confirmed that difference between coefficients of the 
formula (53) is clearly larger, and the formula (53) is matrix to increase noise. 

Here, a practical example for adaptively determining linear matrix 
coefficients M on the basis of setting of ISO sensitivity of the image pick-^p 
apparatus will be described. The image pick-^ip apparatus serves to amplify 
or attenuate signal inputted from the CCD image pick-up device (hereinafter 
referred to as input signal) on the basis of setting of the ISO sensitivity. 
When setting of the ISO sensitivity of the image pick-up apparatus is changed 
from ISO 100 to ISO200, an input signal is amplified so that its value becomes 
equal to, e.g., value which is twice greater than at the time of ISO 100. The 
input signal thus amplified is inputted to the image pick-up apparatus. 
However, in the image pick-up apparatus, since linear matrix processing is 
performed by using the same linear matrix coefficients M with respect to all 
input signals in spite of setting state of the ISO sensitivity, in the case where 
setting of the ISO sensitivity is high, noise component included in an input 
signal is together amplified followed by amplification of the input signal. 
Accordingly, even when attempt is made to raise setting of the ISO sensitivity 



63 



to obtain image of high resolution, an image in which ampHfied noise 
component is included would be generated. 

In view of the above, in the image picknup apparatus according to the 
present invention of this Application, linear matrix coefficients M are 
determined by taking into consideration reduction characteristic of noise 
component included in an input signal amplified or attenuated on the basis of 
setting of ISO sensitivity to perform linear matrix processing by using the 
linear matrix coefficients M. For example, as shown in the Table 1, 
weighting coefficient (wn) with respect to noise reduction characteristic is 
permitted to be changed in accordance with setting of ISO sensitivity to 
substitute wc and wn into the formula (50) and to determine linear matrix 
coefficients M every setting of the ISO sensitivity. Accordingly, since the 
image pick-up apparatus can perform linear matrix processing by using linear 
matrix coefficients M determined on the basis of setting state of the ISO 
sensitivity, even if setting of the ISO sensitivity is caused to be high, there is 
no possibility that noise component is amplified followed by this, thus to have 
ability to obtain image of high resolution. 
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Table 1 



SETTING OF 


w c 


wn 


ISO 






SENSITIVITY 






100 


1 


1 


200 


1 


2 


400 


1 


4 


800 


1 


6 



Here, an example for adaptively determining linear matrix coefficients 
M on the basis of the environment where image of object is picked up by the 
image pick-up apparatus will be described. For example, in the case where 
image of object, e.g., night view, etc. is picked by the image pick-up apparatus, 
there are instances where the dark portion may occupy the large part of 
generated image. When noise takes place at that dark portion, such noise 
becomes very conspicuous. In such a case, it is better to take into 
consideration noise reduction characteristic of noise component rather than 
color reproduction characteristic. 

In view of the above, in the image pick-up apparatus according to the 
present invention, linear matrix coefficients M are determined by taking noise 
reduction characteristic and color reproduction characteristic into 
consideration on the basis of scene in which image of object is picked up to 
perform linear matrix processing by using the linear matrix coefficients M. 
For example, as shown in the histogram of FIG. 23, when 70% of luminance 
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component of generated image or more is included within one half or less of 
the luminance dynamic range of the image pickup apparatus, importance is 
attached to noise reduction characteristic to determine linear matrix 
coefficients M. When except for the above, color reproduction characteristic 
and noise reduction characteristic are taken into consideration to determine 
linear matrix coefficients M. In concrete terms, as shown in the Table 2, 
weighting coefficient (wn) with respect to noise reduction characteristic is 
permitted to be changed in accordance with image pick-up scene to substitute 
wc and wn into the formula (50) to determine linear matrix coefficients M 
every image pick-up scene. Accordingly, since the image pick-up apparatus 
can perform linear matrix processing by using linear matrix coefficients M 
determined on the basis of image pick-up scene, even if dark portion occupies 
the large part of generated image, it becomes possible to prevent noise 
component from becoming conspicuous. 



Table 2 



IMAGE 


w c 


wn 


PICK-UP 






SCENE 






NIGHT VIEW 


1 


8 


OTHERS 


1 


2 



Here, practical example for adaptively determining linear matrix 
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coefficients M on the basis of request of user who uses the image pick-up 
apparatus will be described. There are many instances where image generated 
by picking up image of object by the image pick-up apparatus is required that 
noise is caused to be lesser as compared to color reproduction characteristic 
according to use purpose of user. The use purpose is the matter that 
manufacturing companies (makers) of the image pick-up apparatus do not 
know, but is the fact that only users know. 

In view of the above, the image pick-up apparatus according to the 
present invention of this Application serves to determine linear matrix 
coefficients M on the basis of the condition that user intends, and to perform 
linear matrix processing by using the linear matrix coefficients M. For 
example, as shown in the Table 3, weighting coefficient (wn) with respect to 
noise reduction characteristic is permitted to be changed in accordance with 
noise quantity adjustment variable to substitute wn and wc into the formula 
(50), to determine linear matrix coefficients M every noise quantity 
adjustment variable, and to store the linear matrix coefficient M thus 
determined, whereby when user changes noise quantity adjustment variable 
through the interface of the image pick-up apparatus, predetermined linear 
matrix coefficients M are determined to perform linear matrix processing by 
using the linear matrix coefficients M. Accordingly, since the image pick-up 
apparatus can perform linear matrix processing by using the determined linear 
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matrix coefficients M in accordance with request of user, it is possible to 
perform noise quantity adjustment corresponding to user situation of user 



Table 3 



NOISE QUANTITY 


w c 


wn 


ADJUSTMENT 






VARIABLE 






0 


1 


1 


1 


1 


2 


2 


1 


3 


3 


1 


4 


4 


1 


5 


5 


1 


6 



It is to be noted that the Table 3 means that the larger becomes 
numeric value of the noise quantity adjustment variable, the more noise 
quantity is reduced. Moreover, in place of performing setting of noise quantity 
adjustment variable before image pick-up operation, there may be employed 
an approach in which when image pick-up start button of the image pick-up 
apparatus is pushed down, image after undergone linear matrix processing 
using several linear matrix coefficients M are preserved (stored) into plural 
memories. 

In the image pick-up apparatus constituted in this way, since color 
filter 61 for four colors formed at the front stage unit of the image sensor 45 in 
accordance with the flowchart shown in FIG. 12, it is possible to improve 
"discriminating characteristic of color" of "color reproduction characteristic". 
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Moreover, since matrix processing is performed with respect to signals (R, Gl, 
G2, B) in which luminance can be represented by linear transform at signal 
processing unit 71, color having higher fidelity from a viewpoint of color 
engineering can be reproduced as compared to the case where matrix 
processing is performed with respect to signal obtained after undergone 
gamma processing as in the case of the processing at the signal processing 
unit 11 shown in FIG. 2. Further, since determination of linear matrix 
coefficients M is performed in accordance with the image pick-up condition, 
etc. in the linear matrix processing performed at the linear matrix processing 
unit 95 of the signal processing unit 71, it is possible to improve color 
reproduction characteristic and reduction characteristic of noise of picked up 
images. 

It is to be noted that the present invention has been described in 
accordance with certain preferred embodiments thereof illustrated in the 
accompanying drawings and described in detail, it should be understood by 
those ordinarily skilled in the art that the invention is not limited to 
embodiments, but various modifications, alternative constructions or 
equivalents can be implemented without departing from the scope and spirit of 
the present invention as set forth by appended claims. 

Industrial Applicability 
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Since the image pick-up apparatus according to the present invention 
comprises adjustment unit for adjusting color reproduction value representing 
high fidelity reproduction of color with respect to way of seeing of the eye of 
the human being and noise value representing noise feeling that the human 
being feels, and matrix coefficient determination unit for determining matrix 
coefficients on the basis of adjustment of the adjustment unit, and matrix 
transform processing unit for performing matrix transform processing with 
respect to image which has been picked up at image pick-up device unit 
provided at the image pick-up apparatus on the basis of the matrix coefficients, 
linear matrix coefficients M are adaptively determined in accordance with 
image pick-^p environment and image pickup condition, thus making it 
possible to perform linear matrix processing by using the linear matrix 
coefficients M. 

In addition, since the image pick-up method according to the present 
invention includes first step of adjusting color reproduction value representing 
high fidelity reproduction of color with respect to seeing of the eye of the 
human being and noise feeling that the human being feels, second step of 
determining matrix coefficients on the basis of adjustment of the first step, 
and third step of performing matrix transform processing with respect to 
image which has been picked up at image pick-up device unit which serves to 
pick up image of object on the basis of the matrix coefficients, linear matrix 



